Reinforcement Learning for Solving Stochastic Vehicle Routing Problem

This study addresses a gap in the utilization of Reinforcement Learning (RL) and Machine Learning (ML) techniques in solving the Stochastic Vehicle Routing Problem (SVRP) that involves the challenging task of optimizing vehicle routes under uncertain conditions. We propose a novel end-to-end framework that comprehensively addresses the key sources of stochasticity in SVRP and utilizes an RL agent with a simple yet effective architecture and a tailored training method. Through comparative analysis, our proposed model demonstrates superior performance compared to a widely adopted state-of-the-art metaheuristic, achieving a significant 3.43% reduction in travel costs. Furthermore, the model exhibits robustness across diverse SVRP settings, highlighting its adaptability and ability to learn optimal routing strategies in varying environments. The publicly available implementation of our framework serves as a valuable resource for future research endeavors aimed at advancing RL-based solutions for SVRP.Comment: 14 pages, accepted to ACML2

Quality of Service (QoS) oriented management system in 5G cloud enabled RAN

[EN] This paper analyze techniques to implement QoS/QoE on multi-tenant 5G networks. Describes the ar- chitecture of the next generation mobile network based on cloud-enabled small cell deployments and also proposes an hybrid-cloud solution coexisting with centralized cloud RAN(C-RAN), in order to achieve a gradual implementation of the technology. In this context, the work here presented deals with the challenges of preserving the quality of ex- perience in a multi-tenant cloud enable RAN bearing in mind the Key Performance Indicator(KPI) agreed in the SLA. To achieve this goal, QoS should be managed at different levels of the architecture. Feedback should be given between learning modules in order to analyze the results and infer enhanced decision rules which may conclude in an architecture replacement.The research leading to these results has been supported by the EU funded H2020 5G-PPP projects SESAME (Grant Agreement n 671596) and ESSENCE project (Grant Agreement no 761592) and by the Spanish Ministerio de Economia y Competitividad (MINECO) under grant TEC2016-80090-C2-2-R (5RANVIR).Solozabal, R.; Fajardo, JO.; Blanco, B.; Liberal, F. (2018). Quality of Service (QoS) oriented management system in 5G cloud enabled RAN. En XIII Jornadas de Ingeniería telemática (JITEL 2017). Libro de actas. Editorial Universitat Politècnica de València. 170-175. https://doi.org/10.4995/JITEL2017.2017.6587OCS17017

Regularization of the policy updates for stabilizing Mean Field Games

This work studies non-cooperative Multi-Agent Reinforcement Learning (MARL) where multiple agents interact in the same environment and whose goal is to maximize the individual returns. Challenges arise when scaling up the number of agents due to the resultant non-stationarity that the many agents introduce. In order to address this issue, Mean Field Games (MFG) rely on the symmetry and homogeneity assumptions to approximate games with very large populations. Recently, deep Reinforcement Learning has been used to scale MFG to games with larger number of states. Current methods rely on smoothing techniques such as averaging the q-values or the updates on the mean-field distribution. This work presents a different approach to stabilize the learning based on proximal updates on the mean-field policy. We name our algorithm Mean Field Proximal Policy Optimization (MF-PPO), and we empirically show the effectiveness of our method in the OpenSpiel framework

A cloud-enabled small cell architecture in 5G networks for broadcast/multicast services

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.The evolution of 5G suggests that communication networks become sufficiently flexible to handle a wide variety of network services from various domains. The virtualization of small cells as envisaged by 5G, allows enhanced mobile edge computing capabilities, thus enabling network service deployment and management near the end user. This paper presents a cloud-enabled small cell architecture for 5G networks developed within the 5G-ESSENCE project. This paper also presents the conformity of the proposed architecture to the evolving 5G radio resource management architecture. Furthermore, it examines the inclusion of an edge enabler to support a variety of virtual network functions in 5G networks. Next, the improvement of specific key performance indicators in a public safety use case is evaluated. Finally, the performance of a 5G enabled evolved multimedia broadcast multicast services service is evaluated.Peer ReviewedPostprint (author's final draft

A deep neural network for oxidative coupling of methane trained on high-throughput experimental data

In this work, we develop a deep neural network model for the reaction rate of oxidative coupling of methane from published high-throughput experimental catalysis data. A neural network is formulated so that the rate model satisfies the plug flow reactor design equation. The model is then employed to understand the variation of reactant and product composition within the reactor for the reference catalyst Mn–Na _2 WO _4 /SiO _2 at different temperatures and to identify new catalysts and combinations of known catalysts that would increase yield and selectivity relative to the reference catalyst. The model revealed that methane is converted in the first half of the catalyst bed, while the second part largely consolidates the products (i.e. increases ethylene to ethane ratio). A screening study of ${\geqslant}3400$ combinations of pairs of previously studied catalysts of the form M1(M2) $_{1-2}$ M3O _x /support (where M1, M2 and M3 are metals) revealed that a reactor configuration comprising two sequential catalyst beds leads to synergistic effects resulting in increased yield of C _2 compared to the reference catalyst at identical conditions and contact time. Finally, an expanded screening study of 7400 combinations (comprising previously studied metals but with several new permutations) revealed multiple catalyst choices with enhanced yields of C _2 products. This study demonstrates the value of learning a deep neural network model for the instantaneous reaction rate directly from high-throughput data and represents a first step in constraining a data-driven reaction model to satisfy domain information

Management of mission critical public safety applications: the 5G ESSENCE Project

To address the limitations of legacy Public Safety (PS) solutions, as narrow bandwidth, high deployment cost and poor flexibility, the 5G cellular systems have been proposed. The architecture proposed by the 5G ESSENCE project is based on a cloud-enabled small cell infrastructure with a fully distributed orchestration architecture leveraging multi-access technologies in 5G. Furthermore, SDN and NFV are exploited, with MEC, to create flexible slices for dedicated mission critical public safety applications also at the edge of the network. This is shown by describing as the mission critical push-to-talk services have been implemented in a real testbed.Peer ReviewedPostprint (published version

Management of mission critical public safety applications: the 5G ESSENCE Project

To address the limitations of legacy Public Safety (PS) solutions, as narrow bandwidth, high deployment cost and poor flexibility, the 5G cellular systems have been proposed. The architecture proposed by the 5G ESSENCE project is based on a cloud-enabled small cell infrastructure with a fully distributed orchestration architecture leveraging multi-access technologies in 5G. Furthermore, SDN and NFV are exploited, with MEC, to create flexible slices for dedicated mission critical public safety applications also at the edge of the network. This is shown by describing as the mission critical push-to-talk services have been implemented in a real testbed.Peer Reviewe

Supporting mission critical services through radio access network slicing

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.While the support of Mission Critical (MC) communications on commercial cellular networks has already been incorporated in the latest releases of Long Term Evolution (LTE), it is expected that the network slicing feature of Fifth Generation (5G) systems will further boost the provision of these services thanks to the possibility of creating customized and isolated network slices adapted to the specific requirements of MC communications. At the Radio Access Network (RAN), the realization of a network slice requires to specify how the pool of available radio resources is split between the different slices in accordance with their service requirements. In this context, this paper addresses the use of RAN slicing for provisioning MC services taking as a reference the emergency scenario defined by the 5G ESSENCE project. It is characterized by different stages associated to the occurrence of an incident and its evolution, thus involving different communication needs. For each stage, an estimation of the capacity requirements to be granted to the MC RAN slice is provided. Then, the architecture of the project is discussed, focusing on the components that enable the RAN slicing management to properly support MC services.Peer Reviewe